A Confirmatory Factor Analytic Study Examining the Dimensionality of Educational Achievement Tests

Along with the increasing popularity of item response theory (IRT) in testing practices, it is important to check a fundamental assumption of most of the popular IRT models, which is unidimensionality. Nevertheless, it is hard for educational and psychological tests to be strictly unidimensional. The tests studied in this paper are from a standardized high-stake testing program. They feature potential multidimensionality by presenting various item types and item sets. Confirmatory factor analyses with one-factor and bifactor models, and based on both linear structural equation modeling approach and nonlinear IRT approach were conducted. The competing models were compared and the implications of the bifactor model for checking essential unidimensionality were discussed. Examining Test Dimensionality 3 A Confirmatory Factor Analytic Study Examining the Dimensionality of Educational Achievement Tests Item response theory (IRT) and its applications has been widely used in various educational and psychological testing practices, including test construction, ability estimation, score reporting, equating, and computer adaptive testing. One of the fundamental assumptions for the most popular IRT models is unidimensionality (Hambleton, Swaminathan, & Rogers, 1991), namely only one single latent variable accounts for the item responses. Technically, it can be defined that item responses are independent when a single latent variable is controlled for. To justify the uses of IRT, unidimensionality needs to be demonstrated before any unidimensional IRT modeling is applied. Applying IRT models without the justification of unidimensionality may greatly undermine the advantages of IRT and result in negative consequences. However, it is more often than not that the educational tests are not strictly unidimensional. This could be due to factors such as content variety, construct complexity, and varying item formats. For most tests, a dominant factor rather than only one factor is the goal. The dimensionality analysis described in this paper is therefore to check whether the item responses are essentially unidimensional or “unidimensional enough” for proper IRT applications. Through years of research, a number of different methods have been developed to assess test dimensionality. The main approaches include linear and non-linear factor analysis, nonparametric procedures, multidimensional scaling and structural equation modeling, which is a type of linear factor analysis. In view of the importance of essential unidimensionality, this paper adopted different methods and approaches, including the parallel analysis and bifactor analysis suggested in an earlier paper (see Deng & Hambleton, 2007) and applied to an educational achievement testing program. These tests feature potential multidimensionality by presenting Examining Test Dimensionality 4 various item types and item sets. The purposes of the study were to (1) assess the test dimensionality in both exploratory and confirmatory ways, (2) check the item type effect and check essential unidimensionality, and (3) apply parallel analysis and bifactor analysis methods, and discuss their implications.